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Progress  Report  for  Grant  ONR  N00014-90-J-1114 

"Pulse  Coded  Biologically  Motivated  Neural-Type  MOS  Circuits" 
For  the  Period  05/01/91  -  10/01/91 

1 .  Introduction 

This  project  has  two  aspects,  one  for  ONR  and  one  for  AFOSR.  The 
ONR  portion  is  devoted  to  obtaining  hardware  implementations  -for  the 
physiological  representations  used  in  the  program  SYNETSIM  developed 
by  the  neurophysiologist  Dr.  D.  Hartline  of  Bekesy  Laboratories.  The 
AFOSR  portion  is  for  evaluation  capabilities  of  the  pulse  coded  philos¬ 
ophy  of  neural  networks . 

During  the  period  covered  by  this  report  a  number  of  results  have 
been  obtained,  the  following  being  the  main  accomplishments.  On  the  ONR 
portion  of  the  research  several  chips  have  been  fabricated  for  SYNETSIM 
pools  and  a  neural  arithmetic  unit  based  upon  the  pools,  while  also  a 
number  of  modifications  have  been  made  to  SYNETSIM  to  make  it  a  much 
more  friendly  user  program.  Several  papers  -have  been  presented  at 
international  conferences  and  the  DRIVER  module  continues  rnder  inves¬ 
tigation  for  VLSI  realization.  Mp ’ns  to  implement  long  term  potentia¬ 
tion  also  continue  under  investi  ation.  On  the  AFOSR  portion,  a  signif¬ 
icant  result  has  been  obtained,  ...is  being  a  means  of  realizing  any 
Hopf  ield-type  network  via  pulse  ''t  d  circuits  has  been  obtained  and  a 
paper  on  the  result  is  under  preparation. 

During  this  period  the  PI  and  Co-PI  attended  a  special  meeting  on 
the  decade  of  the  neuron  which  concentrated  on  the  status  of  neural 
studies  throughout  the  world  with  particular  reference  to  cell  func¬ 
tioning.  From  this  a  number  of  new  ideas  were  incorporated  into  the 
research  especially  in  the  area  of  LTP  activity  and  second  messenger 
behavior. 

Participating  in  the  research  during  this  period  have  been: 

a.  R.  Newcomb,  PI 

b.  N.  El-Leithy,  Co-PI 

c.  S.  W.  Tsay,  RA  -  ONR  portion 

d.  M.  de  Savigny,  RA  -  AFOSR  portion 

In  addition  several  other  graduate  and  undergraduate  students  continue 
to  participate  through  independent  study  projects  on  related  topics 
which  enhance  the  research. 

2.  Research  Results  for  the  Period 

2-A.  ONR  Aspects 

In  order  to  design  further  integrated  circuits  which  mimic 
the  compartments  of  SYNETSIM  we  have  gone  into  its  code  in  considerable 
depth  during  this  period.  As  a  consequence  a  number  of  bugs  were  dis¬ 
covered  and  corrected  and  its  capabilities  greatly  enhances  by  placing 
much  of  into  modular  form.  This  allows  it  to  make  use  of  extended  mem¬ 
ory  and  prepares  it  for  transfer  to  a  windows  compatible  progreun  which 
can  run  in  the  background.  The  DRIVER  module  has  been  studied  in  more 
depth  and  to  the  point  that  we  are  now  able  to  characterize  it  by  func¬ 
tions  that  are  realizable  by  BiCMOS  components;  further  research  is 
planned  to  use  approximation  functions  realizable  by  CMOS  circuits. 

A  key  aspect  of  this  research  has  been  our  investigation  of 
the  SYNETSIM  pools  which  give  second  messenger  responses.  Consequently, 
chips  have  been  fabricated  which  realize  the  pools;  these  have  now  been 
received  and  are  under  test.  The  pools  have  led  to  a  very  interesting 
"neural  arithmetic  unit"  (NAU)  which  performs  a  number  of  analog  signal 
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processing  functions,  such  as  multiplication,  division,  and  addition, 
with  very  little  hardware.  Investigation  of  the  first  form  of  the  NAU 
has  led  to  improvements  and  to  a  very  simple  VLSI  realizable  NAU  for 
which  chip  fabrication  has  also  been  undertaken. 

Finally,  studies  continue  on  realization  of long  term  poten¬ 
tiation,  LTP.  The  schema  for  developing  LTP  uses  ideas  stemming  from  a 
combination  of  SYNNETSIM  and  our  former  neural  type  microsystems . 
Consequently,  a  new  type  of  hardware  is  under  consideration  which  com¬ 
bines  the  good  features  of  SYNETSIM  with  those  of  our  neural-type 
cells.  This  should  lead  to  a  simple  system  efficiently  realizable  in 
VLSI  which  can  incorporate  the  complicated  functioning  arising  from 
SYNETSIM  as  well  as  LTP. 

2-B.  AFOSR  Aspects 

During  this  period  a  pulse  coded  realization  of  any  Hopfield 
type  of  neural  network  has  been  developed .  improvements  have  been  made 
in  the  pulse  coded  neuron  which  realizes  all  16  binary  valued  logic 
functions  of  two  inputs.  Because  the  original  circuit  is  inconvenient 
for  VLSI,  a  study  has  been  made  of  how  to  delete  the  filters  used  in 
the  previously  reported  neuron.  From  this  we  have  been  led  to  a  new 
pulse  coded  neuron  which  shows  promise  for  realizing  any  Hopfield  type 
of  neural  network  in  the  pulse  coded  framework.  At  this  point  the  new 
pulse  coded  neuron  has  been  shown  to  allow  us  to  also  generate  14  of 
the  16  binary  valued  logic  functions  without  the  use  of  the  inconve¬ 
nient  filters  of  the  previous  pulse  coded  neuron.  Also  using  the  new 
neuron  a  two  input  MAXNET  has  been  realized.  A  major  problem  in  imple¬ 
menting  Hopfield  types  of  neural  networks  in  the  pulse  coded  philosophy 
is  how  to  encode  general  weights.  A  promising  technique  to  solve  this 
problem  is  under  investigation. 

3.  Publications  and  Theses 

A.  PublicatiorTs 

During  Oiis  period  the  first  of  the  following  papers  were 
presented  at  international  meetings  for  which  they  were  also  published 
in  the  conference  proceedings,  as  noted.  Several  other  papers  are  in 
various  stages  of  preparation. 

a)  S.-W.  Tsay  and  R.  W.  Newcomb,  "A  Neural-Type  Pool  Arithmetic 
Unit,"  Proceedings  of  the  1990  IEEE  International  Symposium  on  Circuits 
and  Systems,  Singapore,  June  1991,  pp.  2518  -  2521. 

^  b)  S.  W.  Tsay,  L.  Chen,  N.  El-Leithy,  G.  Wolodkin,  and  R.  Newcomb, 
"Realization  of  the  Driver  Neural  Network  Module,"  Proceedings  of  the 
IV  International  Symposium  on  Biomedical  Engineering,  Penlscola, 
Spain,  September  1991,  pp.  480  -  481. 

4 .  Research  Assistant  Reports 

Attached  are  concise  reports  from  the  two  Research  Assis¬ 
tants  , 

5 .  Plans  for  Next  Period 

We  plan  to  continue  to  improve  the  capabilities  of  SYNETSIM  by 
making  it  available  to  windows  so  that  multitasking  can  be  undertaken. 
In  line  with  this  we  also  plan  to  further  modularize  SYNETSIM  which 
will  also  allow  us  to  design  CMOS  circuits  to  go  with  the  stand  alone 
modules.  Measurements  on  the  chips  that  have  recently  been  received 
from  the  IC  foundary  (MOSIS)  are  planned  to  be  further  tested  and  rede¬ 
signs  undertaken  to  improve  their  performance. 

The  neural  arithmetic  unit  has  proven  of  considerable  value  and. 


05/01/91  page  3 

thus,  its  continued  improvement  is  to  be  undertaken.  Since  it  appears 
to  give  a  very  practical  means  of  adjusting  weights  in  hardware  inves¬ 
tigation  of  its  use  in  making  electronically  adaptable  neural  networks 
is  planned.  A  major  effort  will  go  into  a  further  development  of  the 
theory  for  LTP,  its  realization  in  SYNETSIM  form,  and  its  design  in  MOS 
circuits.  Because  of  the  significance  of  realizing  Hopfield  type  neural 
networks  in  pulse  coded  form,  further  effort  wil],  be  undertaken  -to  make 
more  rigorous  the  present  results  and  to  prepare  a  publication  in  depth 
on  the  topic.  Several  papers  have  been  accepted  for  meetings  and  these 
will  be  presented  during  the  next  six  month  period. 
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Research  Report 

Research  Assistant;  Suan-Wei  Tsay 


Our  research  works  have  emphasized  five  aspects  during  the  period  from  5/1/90  to 
10/1/91.  These  are  1)  Simplify  the  Neural  Arithmetic  Unit  (NAU)  circuit  and  use  it  to 
design  a  circuit  which  is  equivalent  to  an  adjustable  threshold  MOSFET,  2)  Design 
another  circuit  to  extend  the  functions  of  NAU,  3)  A  paper  entitled  "An  All  MOS 
Neural-Type  Cell"  is  accepted  and  to  appear  in  the  Proceedings  of  the  34th  Midwest 
Symposium  on  Circuits  and  Systems,  Monterey,  CA,  May,  1991,  4)  A  paper  entitled 
"Realization  of  the  Driver  Neural  Network  Module"  is  accepted  and  to  appear  in  the  IV 
International  Symposium  on  Biomedical  Engineering,  Peniscola,  Spain,  September,  1991, 
and  5)  Two  chips  have  been  fabricated  by  MOSIS,  one  for  the  pools  of  types  3  and  4, 
and  one  for  the  NAU.  Parts  1)  and  2)  are  discussed  below: 

1.  Simplified  NAU  and  Adjustable  threshold  MOSFET 

The  previous  version  of  the  neural  arithmetic  unit  originated  from  the  structure  of 
chemical  pools  and  is  shown  in  Figure  1.  The  output  voltage  is  taken  at  node  A/„.  This 
circuit  can  perform  voltage  addition,  voltage  subtraction,  and  voltage  sign  inversion  when 
the  four  nodes  a,  b,  c,  and  d  are  properly  connected.  However,  this  circuit  contains  two 
differential  amplifiers  and  will  occupy  a  chip  area  larger  than  desired. 


c 


(a) 


Figure  T.  A  pool  circuit,  the  background  circuit  for  a  neural  arithmetic  unit,  (a)  The 
intrinsic  circuit  of  a  chemical  pool,  (b)  The  pool  for  inward  and  outward 
currents  being  linearly  modulated  by  other  signals.  Ng  is  the  output  node 
with  output  voltage  P. 

This  NAU  can  be  simplified  to  the  circuit  of  Figure  2  in  which  the  new  NAU  in 
Fig.  2(b)  will  only  occupy  about  half  of  the  area  of  the  previous  version.  Although  circuit 
is  simplified,  the  simplified  NAU  performs  as  well  as  the  previous  NAU. 
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Figure  2:  (a)  Simplified  NAU.  (b)  Actual  circuit  for  an  NAU. 

Utilizing  the  NAU  as  a  voltage  adder,  we  can  make  an  adjustable  threshold 
MOSFET  whose  threshold  voltage  is  modulated  by  the  control  voltage  V^.  The  left-hand 
part  of  Figure  3(a)  is  a  voltage  adder  whose  output  voltage  is  equal  to  the  sum  of  input 
voltages  Vj  and  Vg.  That  is, 

(1) 


We  can  use  Fig.  3(a)  to  define  a  new  4-terminal  device  of  terminals  c,  d,  g,  and  s, 
symbolized  by  Fig.  3(b)  and  called  an  adjustable  threshold  MOSFET.  If  we  assume  the 
threshold  voltage  of  transistor  Mj  to  be  V/  and  the  effective  gate-to-source  voltage  to  be 

the  gate-to-source  voltage  minus  its  threshold  voltage,  the  effective  gate-to-source  voltage 
of  an  adjustable  threshold  MOSFET  will  be 


V  '-V  -V'  =  V  -{-V  -V  -V'  =  V  -  V  -V 


Equation  (2)  defines  V,  in  terms  of  V^',  the  voltage  on  the  gate  of  M,  as  shown  in  Fig. 


3(a).  The  threshold  voltage  for  the  four  terminal  element  of  Fig.  3(b)  is,  consequently. 


y.=y:-y. 


A  PSPICE  simulation  of  this  adjustable  threshold  n-type  MOSFET  is  shown  in  Fig. 
4.  The  six  curves  represent  the  drain  cun'ents  with  the  control  voltage  swept  from  OV 
to  IV  in  0.2V  steps.  The  effective  threshold  voltages  are  shown  to  be  about  IV  when  V^ 
=  OV  and  about  OV  when  V^  =  IV  and  vary  almost  linearly  between  these  ranges. 
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Figure  3;  An  adjustable  threshold  n-type  MOSFET.  (a)  The  gate  voltage  of  M,  is  tied 
to  the  output  of  a  voltage  adder  so  that  the  threshold  voltage  can  be  mod¬ 
ulated  by  the  contiol  voltage  V^.  (b)  The  symbol  for  an  adjustable  tJireshold 
n-type  MOSFET. 

Adju&ubit  Thfc&hokJ  MOSI'ET 


Figure  4:  Simulation  of  an  adjustable  tlireshold  n-type  MOSFET  using  an  adder  from 

NAU. 
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Root 

When  we  replace  one  of  the  differential  pairs  in  NAU  by  the  circuit  of  a  multiplier, 
we  can  make  another  arithmetic  unit  that  performs  multiplication,  division,  square,  and 
square  root.  In  Figure  5,  the  multiplier,  represented  by  a  rectangular  box,  is  implernented 
by  a  MOS  Gilbert  multiplier.  We  tlian  have 

A=/,-/,  =  CKV,  (4) 

=  (5) 

where  C  and  C’  are  constants. 


Figure  5:  A  modified  NAU  that  performs  multiplication,  division,  square,  and  square 

root. 

The  circuit  of  Fig.  5  reaches  its  equilibrium  state  when  /,  equals  4.  That  is 

(6) 

Q 

where  A"  =  —  is  a  constant.  According  to  equation  (6)  we  can  use  this  circuit  to  perfomi 
different  functions  as  summarized  in  Table  1. 
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Function 

V. 

■■ 

Vc 

y. 

Result 

Multiplication 

P 

GND 

V. 

mm 

P=KV,V, 

Division 

Vu 

GND 

p 

y. 

’ 

P  =- 

Square 

P 

GND 

B 

P=KV- 

Square  Root 

y. 

GND 

P 

p 

P=y[K^, 

Table  1:  Input  assignments  for  the  NAU  (Figure  5)  to  perform  multiplication, 

division,  square,  and  square  root. 
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Research  Assistant:  Marc  de  Savigny 
Period  5/1/91  to  10/1/91 


Summary: 

Research  performed  during  the  last  six  months  has  included  additional  work 
of  a  pulse  coded  Hopfield  neural  network.  In  order  to  decrease  the  size  of  each 
neuron,  small  VLSI  integrable  capacitance  multipliers  have  been  studied  and  sim¬ 
ulated.  A  living  animal’s  behavior  was  simplified  to  be  modeled  completely.  We 
should  now  be  in  a  position  to  realize  a  small  anificial  brain,  as  we  have  both  the 
constituting  elements,  and  how  they  interact. 

Hopfield  pulse  coded  neural  network: 

Research  on  pulse  coded  Hopfield  neural  networks  is  well  advanced.  A  paper 
consigning  the  results  of  our  investigations  is  underway. 

The  major  contributions  is  that  we  show  that  pulse  coded  neural  networks  can 
be  implemented  under  Hopfield’s  form.  Also,  examples  for  both  analog  and  digital 
operations  are  given.  In  the  last  report,  we  said  that  we  needed  to  limit  the  inputs 
to  constant  levels.  This  is  no  longer  the  case.  A  small  modification  of  the  pro¬ 
cessing  element  has  removed  this  limitation.  Also,  a  way  to  adjust  the  weights  on 
chip  is  given. 

Capacitance  multipliers: 

When  implementing  circuits  in  a  VLSI  form,  capacitors  and  resistors  of  large 
value  (more  than  a  few  pF,  dozens  of  kohms,  respectively)  occupy  a  considerable 
space  on  a  chip.  This  is  one  of  the  major  drawback  of  VLSI  implementation.  As 
a  consequence,  either  large-valued  passive  components  must  be  avoided,  or  very 
few  devices  can  be  included  in  a  given  chip.  The  problem  is  even  more  critical 


when  dealing  with  neural  networks,  since  we  want  to  integrate  a  large  number  of 
identical  processing  elements.  If  we  can  decrease  the  area  needed  for  the  processing 
elements,  we  will  be  able  to  increase  the  size  of  the  network. 

We  have,  therefore,  investigated  devices  that  act  like  capacitors,  but  occupy 
only  a  fraction  of  the  area  needed  by  a  capacitor  of  same  value  implemented  directly. 
We  call  these  devices  capacitance  multipliers.  At  this  point,  we  have  half  a  dozen 
circuits,  which  are  optimized  for  different  additional  criteria.  The  most  promising 
device  shows  an  area  gain  of  about  4000.  Additional  research  is  required  to 
eliminate  remaining  drawbacks  of  this  capacitance  multipliers.  For  example,  the 
capacitors  do  not  discharge  completely,  i.e,  a  voltage  drop  is  always  measured 
across  the  device,  even  if  the  equivalent  capacitor  is  supposed  to  hold  no  charge. 
Also,  we  are  working  on  a  manuscript  for  a  paper  to  present  our  findings. 

Study  of  a  biological  neural  system: 

In  order  to  better  understand  how  biological  neural  networks  affect  the 
behavior  of  living  creatures,  we  have  selected  the  example  of  a  simple  coastal  snail. 
Its  behavior  was  simplified  so  that  it  can  be  explained  by  the  concunent  competing 
action  of  three  basic  feelings:  hunger,  fear,  and  thirst.  The  animal  has  a  collection 
of  sensors  which  deliver  information  from  the  outside  world,  (two  pairs  of  eyes, 
skin,...)  as  well  as  the  state  of  the  animal  (stomach,...).  This  information  is  processed 
to  determine  the  winning  feeling,  and  hence  the  action  to  take,  and  the  command 
to  apply  to  the  muscles.  We  have  written  a  program  in  PASCAL  to  show  that  the 
simulated  animal  behaves  like  a  real  one.  It  was  discovered  that  after  a  transition 
time,  the  simulated  animal’s  behavior  was  periodic.  Like  a  living  creature,  the 
simulated  animal  would  live  in  a  cyclic  fashion  (for  example  eat  and  sleep  peri¬ 
odically),  in  a  way  that  would  never  jeopardize  its  existence.  This  is  largely  what 
happens  in  real  life.  Random  events  that  are  not  under  the  creature  control,  as  well 
as  faculty  impairment  due  to  old  age  were  not  simulated. 

We  now  have  ail  the  elements  to  build  the  snail’s  brain:  the  system’s 
description  is  given  by  the  program  mentioned  above,  and  the  neurons  themselves 
can  be  implemented  by  the  pulse  coded  neurons  studied  above  in  this  research. 


